Numerical Optimization Lecture 13: Conjugate Gradient Method

We still need to show that the directions p0, p1, . . . , pn−1 generated by Algorithms 1 and (2) are conjugate wrt A. If so, then by Theorem 11.3, this algorithm will terminate in n steps. The next theorem shows this property, along with two other important properties: (i) the residuals ri are mutually orthogonal, and (ii) each pk and rk is contained in the Krylov subspace of degree k for r0, defined by K(r0; k) = span{r0, Ar0, . . . , Ar0}. To understand its relation to CG, we can see from the fact that r0 = Ax0 − b and A−1 can be written due to the Cayley-Hamilton theorem, A−1 ≈ (−1) k−1